We're probably going to need a bit of a backoff here. I would recommend using an exponential backoff strategy

At least adding some support for backoff would be nice. But I'm also okay with adding that in a second step.

Creating a new Metrics object for each POST is going to be quite expensive

Yes, it's not ideal; unfortunately, there is no end-to-end scoped instrumentation in CAGG. Not to mention the event loop here is async and so decoupled. If CAGG supports periodic metrics, we could use that, but this is not unreasonable (there's an MF in the Router so for each incoming request; Ref).

Looks like you're missing a test case for the feature you're trying to add. Can you please add a test showing the retry on a failed POST?

Maybe add a base case

    private HttpResponse sendHttpRequest(
            final Request request
    ) { 
      return this.sendHttpRequest(request, 0)
    }

(I recognize there's already a Thread.sleep(1000) in this test case, but - why 3000?)

Because I added exponential backoff for the retry, which means, after initial HTTPrequest, the thread will wait 1000 milliseconds before sending the next attempt. So if we use 1000 here this test will fail.

Sleeps are a really bad way to deal with this and will cause brittle tests. Waiting for an asyc completion with a timeout is probably a better way to go

You can't do a thread sleep here. It can block too much. The rule is basically not to block in an asyc method.

Also, this isn't proper exponential backoff. Please take a look at https://en.m.wikipedia.org/wiki/Exponential_backoff

Specifically the part about randomization.

By saying "block too much", do you mean 1) it can block more than the time set or 2)it occupies too much resources because it is a busy wait?
I found a solution here using ScheduledExecutorService which can solve 1).
And we can probably use wait() for a non busy wait if the problem is 2).

You need to understand how this code is executing. This code is in an Akka actor, which means it is running on a thread pool shared with other actors. There is a limited amount of threads. In an actor, you should not do anything that blocks the thread (in most cases, there can be exceptions if you are very careful and configure things explicitly). But this also gives us access to Akka timers where you can schedule execution in a non blocking way. Please take a look at the following documentation.

https://doc.akka.io/docs/akka/current/actors.html#introduction

https://doc.akka.io/docs/akka/current/dispatchers.html

https://doc.akka.io/docs/akka/current/actors.html#actors-timers

Sleeps are a really bad way to deal with this and will cause brittle tests. Waiting for an asyc completion with a timeout is probably a better way to go